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Trellis Graphics that allows more than three variables to be visualized simultaneously, 
has uncovered startling results that go beyond previous analysis, provide evidence to 
support the anecdotal claims, and show that overall satisfaction is not a reliable 
measurement for determining patient satisfaction. The seven factors defined by the 
National Committee on Quality Assurance are each individually, and together as a group, 
more reliable measures. The inability to choose a provider was clearly rated by every 
beneficiary group as the single greatest source of dissatisfaction. There are also 
differences in satisfaction between the sexes, and among the different groups. Active 
duty members, who are the primary customers of military treatment facilities, are the 


most dissatisfied, and women tend to be less happy than men. 
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EXECUTIVE SUMMARY 


During times of peace, health care is one of the foremost quality of life issues to active 
duty members, their families and retirees. Patient satisfaction surveys are used to 
determine how patients perceive salient aspects of their medical care. There has been 
substantial anecdotal evidence to suggest that patients are unhappy with their care, but 
past analysis of the DOD Annual Surveys using simple frequencies of responses indicated 
that, overall, patients were satisfied. This thesis, using a powerful new technique called 
Trellis Graphics that allows more than three variables to be visualized simultaneously, 
has mncovered startling results that go beyond previous analysis, provide evidence to 
Support the anecdotal claims, and show that overall satisfaction is not a reliable 
measurement for determining patient satisfaction. The seven factors defined by the 
National Committee on Quality Assurance are each individually, and together as a group, 
more reliable measures. The inability to choose a provider was clearly rated by every 
beneficiary group as the single greatest source of dissatisfaction. There are also 
differences in satisfaction between the sexes, and among the different groups. Active 
duty members, who are the primary customers of military treatment facilities, are the 
most dissatisfied, and women tend to be less happy than men. 

This thesis provides a new technique for analysis of complex surveys and attitudinal 
studies that could have far-reaching impact on how surveys are analyzed throughout the 
Department of Defense. Surveys are the central tool used within DOD for measuring 


intangible factors such as attitudes, feelings, and quality of life issues. Almost all 


1X 


surveys in DOD use Likert scales that is, they collect discrete ordinal responses. The 
accepted analysis of an ordinal scale survey is to use simplistic frequency tables or to use 
logistic regression for more advanced work. Logistic regression for the patient 
satisfaction data is unsatisfactory because of the huge models that resulted from the 
complex interactions among the demographic and the measurement variables. At this 
point, after the accepted types of analysis were used and failed, a relatively new graphing 
technique called Trellis Graphics was adapted to understand the complex interactions that 
were occurring within the data. This technique allows the user to look past the three- 
dimensional barrier and visualize the hypervariate data in four and five dimensions. This 
graphics technique had previously been used to analyze agriculture and environmental 
measures, not attitudinal surveys. The results were both clear and useful. One important 
impact of the research found an area of major dissatisfaction among active duty members 
and discovered a possible flaw in the DOD survey design. The most significant impact of 
this research is putting into practice a new method that introduces a fundamentally new 
approach to analysis of survey results, never before considered. Both the analyst and the 
end user can visualize the data, which adds confidence to the decision-maker and 
enhances the overall benefit of the analysis. Trellis graphics could be used by all of the 
military services to analyze and amplify the underlying foundations of attitudes and 
feelings for many of the quality of life issues such as retention, housing, pay, and medical 
care. It is these issues that DOD and the Congress have realized are vitally important to 
preserving today’s peacetime military force at its highest levels of readiness. The 


technique introduced herein, can make a significant contribution to the understanding of 


those issues and ultimately enhance decision maker’s ability to improve the quality of life 


for all members of the military community. 
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I. INTRODUCTION 


The Department of Defense operates one of the nation’s largest healthcare complexes, 
with nearly 8.3 million individuals eligible to receive care through the Military Health 
System. Since the end of the Cold War, the U.S. military has seen manpower reductions, 
which have changed the role of military medicine. Since the Department of Defense 
began its downsizing, the number of doctors, nurses and medical technicians has declined 
by as much as 50 percent in some locations. Furthermore, the Base Realignment and 
Closure Commission (BRAC) recommended closing a number of installations that are no 
longer needed. As a result of this and other downsizing efforts, 35 percent of the military 
hospitals that existed in the U.S. in 1997 have closed. However, the total number of 
people seeking healthcare through the MHS has dropped by only 9 percent. (OSD 1997) 

In order to mitigate the possible decline in the ability to care for its patients, the 
Military Health System (MHS) must examine itself in this environment and move toward 
the standards set by the civilian healthcare industry. These standards are outlined by two 
of the most prominent healthcare accrediting bodies: the Joint Commission of 
Accreditation of Health Care Organization (JCAHO) and the National Committee on 
Quality Assurance (NCQA). One of the most important standards detailed by these 
accrediting organizations and the focus of this paper is the measurement of a patient’s 
satisfaction. In the future when all hospitals, military and civilian, have virtually the 
same kind of healthcare products, the distinguishing factor that will ensure an 
organization’s survival will be its attention to high-value customer services (Nelson 
1991). Therefore, in addition to high-quality healthcare products, patient satisfaction will 


be the key ingredient for an organization to be competitive (Nelson 1991). 


In a recent study for the White House office of Consumer Affairs it was claimed: 

e “96 percent of unhappy customers never complain about rude or discourteous 
treatment, but 

e 90 percent or more who are dissatisfied with the service they receive will not buy 


again or come back. Worse still, 
e Each of those unhappy customers will tell his or her story to at least nine other 


people, and 
e 13 percent of those unhappy former customers will tell their stories to more than 
twenty people.” (Desatnick 1987) 

The U.S. healthcare industry in general, including the MHS, is undergoing sweeping 
changes. The healthcare industry is experiencing a shift in focus from specialty providers 
to medical generalists, from inpatient to outpatient care, and from traditional indemnity 
insurance to prepaid health plans. As the Military Health System moves towards the 
future, its leaders have many challenges facing them. Ensuring customers are satisfied 
with the products from both the military hospitals and the network of civilian providers 
will be the key to providing care for beneficiaries. 

A. PURPOSE AND STRUCTURE OF THESIS 

Patient satisfaction has been shown to be positively correlated to treatment 
compliance, decreasing no-show appointments, better outcomes, and decreased 
disenrollment from health plans (Cleary 1989). The primary objective of this paper is to 
determine if there are key areas such as: 1) access to healthcare, 2) choice of providers 
and continuity of care, 3) perceptions of technical competence of providers, 4) overall 
quality of healthcare, 5) communication between providers and patients, 6) perception of 
being able to get help with financial needs, and 7) interpersonal skills of the providers, 
that may help military medical facilities improve patient satisfaction. A second area of 


interest is whether there are distinct differences in satisfaction between men and women 


in the different regions. Finally, there is interest in whether there are differences in 
satisfaction among the four military beneficiary groups: 1) active duty personnel, 2) 
family members of active duty personnel, 3) retirees, their family members, and survivors 
under age 65, and 4) retirees, their family members, and survivors, age 65 and over. 

Access was of primary importance to each of the four beneficiary groups in the 1994- 
95 Annual Survey. In addition to access, having the freedom to choose one’s doctor and 
location of care may be of great importance to military beneficiaries. As outlined in 
chapter 55 of Title 10 United States Code, the eligibility status of the individual plays a 
role in determining the location of care he or she may obtain from either the civilian 
sector or a military facility, thereby affecting his or her choice of provider. 

This thesis will begin by providing some examples of how other organizations have 
been using customer patient satisfaction surveys. Chapter II gives an introduction to the 
Military Health Systems’ benefit plan and provides an overview of the types of medical 
plans members may choose. Then Chapter III will give a brief description of earlier work 
on patient satisfaction, discuss a working definition of patient satisfaction and, of special 
importance, present an overview on how the National Committee for Quality Assurance 
(NCQA) has defined patient satisfaction. Chapter IV will describe the 1996 Health Care 
Survey of DoD Beneficiaries Form A and the analysis of the survey. Chapter V will 
describe the results of logistic regression and the relatively new visualization technique of 
Trellis Graphics using a subset of the 1996 survey data. And lastly, Chapter VI discusses 


the implications of the analysis and gives conclusions and recommendations. 


B. CUSTOMER SURVEYS 

Customer surveys have been used as a primary tool in both the military and civilian 
markets for assessing whether patients are satisfied with the different aspects of their 
medical care. In the past, patient satisfaction surveys were primarily used by hospitals 
and managed care organizations as marketing tools (Kennedy 1996). Many 
organizations were comfortable with a fairly low response rate, while some organizations 
even appeared to believe that simply giving a patient a survey indicated that the 
organization cared about the patient’s opinion, regardless of whether or not the survey 
results were actually used agence 1996). In today’s competitive environment and with 
healthcare costs rising, hospitals and managed care organizations are now realizing they 
need more reliable and useful data from patient satisfaction surveys. 

Four primary conditions have prompted the change in this direction and redesign of 
surveys. First, and foremost, the Joint Commission of Accreditation of Hospital 
Organizations and the National Committee on Quality Assurance have released standards 
requiring organizations to continuously measure patient satisfaction. These measures are 
to provide data and feedback to both the consumers and purchasers of healthcare. The 
second condition is that healthcare is becoming increasingly competitive and costly 
which leads to a heavy emphasis on the resulting quality of healthcare services. Thirdly, 
the patient’s viewpoint has become much more important and relevant in the adoption of 
continuous quality improvement methods and programs. Lastly, employers as purchasers 
of healthcare are demanding information on satisfaction, costs, doctor/patient relations, 


convenience and accessibility for their employees. (Kennedy 1996) 


Several prominent organizations have used patient satisfaction surveys successfully 
and have been able to link customer judgments with process measures. One example of 
this is the Henry Ford Medical Group, one of the largest group medical practices in the 
United States that specializes in primary care, located in Detroit, Michigan. This 
organization was able to use a survey to identify problems that their customers had with 
access. Two of the major problems identified were that patients could not get through on 
the telephone to obtain one of the limited number of appointments, and that they had to 
wait for long periods of time between scheduling an appointment and seeing the doctor. 
The Group used a survey to ask their patients about key issues that influenced their 
overall satisfaction. The survey results showed that the significant determinants for 
satisfying their patients, were ease of scheduling appointments, being able to see a 
specialist when needed, having a shorter waiting time to see a doctor, and the ability to 
obtain medical advice via phone. (Anctil 1996) 

In another study, a survey was conducted on inpatient care at Bigham and Women’s 
Hospital in Boston. The intent of the study was to seek general information about patient 
satisfaction, and to investigate relationships between overall satisfaction and 
characteristics such as age, gender, health status, source of payment and location in the 
hospital where the patients’ received care. The Brigham and Women’s Hospital found 
that for inpatient care, perceived health and their satisfaction with nurses and physicians 


were the two strongest indicators of patient satisfaction. (Cleary 1989) 





Hi. MILITARY HEALTH SYSTEM 
A. CURRENT HEALTH BENEFTIS PLAN 

The Military Health Services’ primary mission is to maintain the health of its active 
duty service personnel and to be prepared to deliver health care during times of war. In 
addition to its primary mission, the MHS, as an employer, offers health care services to 
non-active duty beneficiaries. Title 10, U.S.C. Chapter 55, Sections 1071-1106 outline 
the entitlement programs for the Uniformed Services and establishes the eligibility 
requirements for the MHS. The healthcare services are provided through a system of 
medical centers, smaller hospitals, and clinics worldwide, and through a DoD- 
administered insurance-like program called the Civilian Health and Medical Program of 
the Uniformed Services (CHAMPUS). CHAMPUS 1s automatically available to families 
of active duty personnel, retirees and their dependents and survivors under the age of 65. 
At age 65, beneficiaries are no longer eligible for CHAMPUS because they become 
eligible for Medicare. (TSO 1997) 

In the 1980’s, MHS faced CHAMPUS costs which grew 350 percent while the 
national health expenditures increased by only 166 percent during the same period. In 
response to the escalating costs, DoD, with congressional authorization, has been testing 
alternative approaches for delivering healthcare such as managed care which emphasizes 
keeping the patient healthy through preventive care and services rather than simply 
treating the patient when he or she becomes ill. (GAO 1995) 

The purpose of converting the MHS to a managed care system was to provide quality 
care while improving access and containing costs. This initiative toward managed care in 


the MHS evolved over several years. The Fiscal Year 1988 Defense Authorization Act 


(Public Law 100-180) initiated this effort by mandating demonstration projects be 
implemented to improve quality and contain costs in the MHS. These projects included 
the CHAMPUS Reform Initiative in Hawaii and California (CRI), Catchment Area 
Management Demonstration Projects, and the U.S. Army’s “Gateway to Care” program 
implemented in 1991. All of these initiatives were developed to contain the rapidly 
rising healthcare costs for CHAMPUS, to maintain or improve the quality and 
accessibility of patient care, and to improve both beneficiary and Military Treatment 
Facility (MTF) staff satisfaction. However, under some of the demonstration projects, 
costs actually increased: in the case of CRI, for example, by 4.6 percent from 1988 to 
1989. The Assistant Secretary of Defense for Health Affairs addressed the demonstration 
projects’ shortcomings by developing TRICARE. (GAO 1995) 
B. TRICARE 

TRICARE itself 1s a combination of these different managed care projects. It was 
established by the Secretary of Defense under the authority of chapter 55 of Title 10, 
United States Code, principally, section 1097. The military health system’s product is a 
benefits package modeled after the civilian sectors’ managed care plans, preferred 
provider plans and fee-for-service plans. TRICARE is a regionally managed healthcare 
program for active duty and retired members of the uniformed services, their families and 
survivors. Jt brings together the healthcare resources of the Army, Air Force, and the 
Navy, and supplements them with networks of civilian healthcare professionals to 
provide access and quality services while maintaining the capability to support military 


operations. (OSD 1997) 


TRICARE features a triple-option benefit, offering beneficiaries eligible for 
CHAMPUS two new options for health care in addition to the CHAMPUS program. The 
options vary in the choices beneficiaries have in selecting their physicians and the 
amount beneficiaries are required to contribute toward the portion of the cost of their care 
received from civilian providers. (OSD 1997) 

1. TRICARE Prime 

The first new option, the TRICARE Prime option, is a HMO-like alternative that 
provides comprehensive medical care to beneficiaries through an integrated network of 
military and contracted civilian providers. Active duty members are automatically 
enrolled in this option and beneficiaries who select this option must enroll annually in the 
program, agreeing to go through an assigned military or civilian primary care physician 
for all of their care. The incentives for enrolling in Prime are: 1) no enrollment fees for 
active duty and their dependents, and low enrollment fees for all others, 2) low co- 
payments, 3) no annual deductible, and 4) no paperwork. Retirees, their dependents, and 
survivors age 65 and over are not eligible to enroll in Pnme but remain eligible to receive 
care in military medical facilities. TRICARE prime is similar to health maintenance 
organization (HMO) programs offered by employers across the U.S. (OSD 1997) 

2. TRICARE Standard 

TRICARE Standard is the current fee-for-service CHAMPUS program. This option 
provides beneficiaries the greatest freedom of choice in selecting civilian providers, but it 
also requires the highest beneficiary costs and is the most expensive way to receive 
healthcare in the MHS. Beneficiaries who use Standard must: 1) meet an annual 


deductible, 2) pay a co-payment of 20 percent for dependents of active duty and 25 


percent for retirees, their dependents, and survivors of active duty members, and 3) be 
responsible for CHAMPUS claims in some cases. There is no requirement to enroll in 
the Standard option, and the patient may still seek care at a military hospital or clinic on a 
space-available basis. (OSD 1997) 

3. TRICARE Extra 

TRICARE Extra is the second new option and lies between TRICARE Prime and 
Standard. It is a “preferred provider plan,” through which beneficiaries receive a 5 
percent discount on their co-payments if they use the civilian contracted network; 
however, they still must first pay their deductible. In most cases, healthcare will be 
more expensive than with TRICARE Prime and less expensive than Standard, but the 
beneficiary will still maintain a moderate degree of freedom of choice of provider. As 
with the Standard option, enrollment is not required. Patients in this option may seek 
care at a military hospital or clinic where space is available, but most of those 


appointments will be reserved for TRICARE Prime enrollees. (OSD 1997) 
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If. LITERATURE REVIEW 

Patient satisfaction means different things to different people. It is important to 
understand it in terms of behavioral models and to give it a working definition. In 1983 
Gregory Pascoe from the University of California did an extensive literature review on 
the behavioral models of patient satisfaction in primary health care settings. The 
behavioral models provide insight into what things such as attitudes, expectations and 
beliefs, make up patient satisfaction, while a working definition provides a practical way 
to define it so 1t can be measured. 
A. BEHAVIORAL MODELS 

In the past, patient satisfaction has been the focus of three different theories 
summarized in the literature: expectancy-value theories, fulfillment theories and 
discrepancy theories. The expectancy-value model, developed by Linder-Pelz in 1982, 
proposed that attitudes that influence expectations are the primary indicators of patient 
satisfaction. Some examples are attitudes about the health care system in general, 
attitudes about the location of care received, and attitudes about the cost of treatment. An 
alternative theory is the fulfillment theory, which defines satisfaction as a function of the 
amount of care and attention received from a situation regardless of how much one feels 
they should receive or how much they would want to receive. The third model 1s the 
discrepancy theory, which includes the patient’s perception of what is expected and 
valued, and which defines satisfaction by the difference between actual outcome and the 
patient’s ideal of what the outcome should be. (Pascoe 1983) 

Although each of these models describes patient satisfaction to some extent, each has 


some area of logical weakness. The expectancy-value model developed by Linder-Pelz is 
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closest to general acceptance. However, it is limited because although satisfaction in 
general has to do with expectations influenced by attitudes, satisfaction by actual services 
received could be influenced more by the patient’s reaction to the actual experience than 
to their initial expectations. On the other hand, the fulfillment model assumes that 
objective outcomes alone will determine patient satisfaction and ignores the 
psychological aspects of defining care and services received. Lastly, the major 
disadvantage of the discrepancy theory model is that it does not take into account the 
reverse implication, which is the assumption that any difference between outcomes will 
result in dissatisfaction even if the actual outcome was better than the patient’s original 
expectation of the outcome. It has been shown that each of these models that attempted 
to define patient satisfaction has some logical weakness and furthermore, none has been 
proven by empirical evidence. Because of these problems, researchers turned to 
consumer Satisfaction models for insight in defining patient satisfaction. (Pascoe 1983) 
Researchers have mostly agreed that consumer satisfaction is multidimensional, 
where overall satisfaction depends on the customer identifying important attnbutes of a 
product or service, and then determining if most of the attributes, or the most important 
attributes, are deemed favorable. The three major models used in customer satisfaction 
are the contrast model, the assimilation model and the assimilation-contrast model, all of 
which use the expectancy approach. The contrast model is based on the magnification of 
differences perceived between expectation and performance by the consumer, while the 
assimilation model goes one step further. In this second model, consumers will adjust 
their perceptions of performance to match their expectations by assimilating the 


difference, and lower their expectations. The hybrid of the above two models is the 


assimilation-contrast model. With this model it is accepted that expectations are the 
customer’s standard of judgement but there is some measurement of assimilation or 
latitude of acceptance if the customer’s exact expectations are not met. All three of these 
models have been empirically tested and supported. (Pascoe 1983) 

A combination of the patient satisfaction models and the consumer satisfaction models 
can be used to conceptualize what patient satisfaction is and what role it plays in 
healthcare. Psychological factors play a role in how patient satisfaction is defined. 
Pascoe defines patient satisfaction as “a healthcare recipient’s reaction to salient aspects 
of the context, process, and result of [his or her] service experience” (1983). Patient 
satisfaction is then a comparison of a subjective standard to the actual healthcare 
experience of the service received. This may be based upon a subjective ideal, a sense of 
what is deserved, an average of past experience in other situations, or upon some learned, 
minimally acceptable level of service. (Pascoe 1983) 

In understanding patient satisfaction the behavioral models give us insight into the 
human factors of attitude and expectations. However, patient satisfaction, in order to be 
measured, must be defined by a working definition. Research has shown that this can be 
done through defining patient satisfaction in terms of several factors. 

B. A WORKING DEFINITION OF PATIENT SATISFACTION 

Throughout the literature, there are some common themes of factors that indicate 
patient satisfaction. These are accessibility, provider-patient interaction, clinical 
outcome, and choice of provider. (Pascoe 1983) 

Access is one of the most important factors in determining patient satisfaction. It has 


many dimensions such as availability, accommodation, affordability, and acceptability 


(Penchansky 1983). In Pascoe’s review of recent studies, he found that all of the studies 
with one exception had shown a significant relationship between increased access and 
patient satisfaction (1983). Dissatisfaction was highly correlated with difficulty in 
scheduling appointments, length of waiting time between appointments, increased 
distance to travel to obtain care, and longer delays in obtaining services (Pascoe 1983). 

The factor of provider-patient interaction has typically included technical competence, 
interpersonal skills of the provider and the length of time the visit lasts. The perception 
of a provider’s competence is strongly interrelated with the provider’s interpersonal skills 
and communication skills. High patient satisfaction has been indicated by a provider’s 
ability to provide clear and adequate explanations, which in turn increase the patient’s 
perception of the provider’s technical competence. (Pascoe 1983) 

A third factor, clinical outcome, has had mixed reviews on whether it is a valuable 
factor. Clinical outcomes are represented by a change in a patient’s health status that is 
directly attributable to the treatment received. Most satisfaction surveys do not account 
for the patient’s condition prior to treatment. In general, however, the severity of illness 
or health status of the patient will play a role in how the patient feels about the care he or 
she has received. A study by Stewart and Wanklin in 1978 found no relationship 
between outcomes and patient satisfaction, but other studies have found small to medium 
correlation with patient satisfaction. (Pascoe 1983) 

The final factor is choice of provider. There has been significant evidence to indicate 
that the provider-patient fit is v-ry important for a patient’s overall satisfaction. For 
example, in a study cited in the journal Hospitals, Kaiser Permanente of Northen 


California surveyed 10,000 beneficiaries and found that choice in choosing one’s own 
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primary care manager was the single most important predictor of overall satisfaction 
(Greene 1998). In astudy by Ross in 1982, it was found that a patient was more likely to 
have increased satisfaction if the patient’s and the physician’s sociodemographics 
matched. Within this dimension of provider-patient fit is also continuity of care. Having 
a regular source of care and seeing the same provider are strongly related to high patient 
satisfaction (Pascoe 1983). 

In addition to behaviors and the above-mentioned factors, two socio-demographic 
categories, sex and age, are often used in studies on patient satisfaction. In general, it 
was found from studies in the late 1970’s that older patients tend to report higher levels 
of satisfaction than do satire: patients and that women tend to be more satisfied than 
men (Pascoe 1983). However, there have been other studies from that same time period 
that have shown younger patients were more satisfied than older patients (Pascoe 1983). 
A survey conducted in 1989 at Brigham and Women’s Hospital in Boston had mixed 
results on whether sex and age are significant in predicting patient satisfaction. Each of 
these variables may play a role in determining satisfaction. However, these could be 
somehow related to the organization, the severity of illness, or other factors (Pascoe 
1983). 

C. HEDIS 

Research seems to show patient satisfaction information does have an important role 
in evaluating primary health care. Because of its importance, the National Committee for 
Quality Assurance (NCQA), an organization whose goal is to report publicly on the 


quality of managed care plans, developed a system for standardizing the measurement of 


salient aspects of quality care. This system is called the Health Plan Employer Data and 
Information Set (HEDIS). 

HEDIS is a report card project that consists of a set of standardized performance 
measures developed by the Committee on Performance Measurement (CPM). The 
committee was made up of representatives of many different populations of constituents 
from both the public and the private healthcare sectors. These include representatives of 
Medicare, Medicaid, organized labor, medical providers, public health officials, health 
plan sponsors and lastly but of utmost importance, technical advisors who brought 
scientific mgor to the process of defining standard performance measures. The original 
purpose of HEDIS was to provide to purchasers and consumers of healthcare information 
based on outcomes to compare managed healthcare plans based on quality and not just 
cost. Eight performance measures were defined by the Committee on Performance 
Measures, one of which is “Member Satisfaction,” the focus of this paper. The 
development of this report card and its performance measures by the CPM was very 
important in standardizing many of the healthcare industry’s views, evaluations, and 
basic definitions. The million-dollar question has always been “what constitutes quality 
healthcare?” Through HEDIS and the efforts of the CPM, the establishment of these 
performance measures has addressed this question. Additionally, these measures have 
been further outlined by a set of defined categories as indicators. (HEDIS 1997) 

The construct that is of prime importance to this paper is the definition of Patient 
Satisfaction as described in the “Member Satisfaction” measurement. This contains 
seven indicators of overall satisfaction which are: 1) Access: the ease in arranging for and 


getting care; 2) Communication: the level of explanation and attention given to the 


patient; 3) Choice and Continuity: the ability of the patient to see the doctor of your 
choice; 4) Interpersonal care: the extent to which the medical staff takes a personal 
interest in the patient and his or her medical problems; 5) Quality and Outcomes: the 
overall quality of care, the outcomes of medical care, and how much the patient is helped; 
6) Technical quality: the thoroughness of examinations and the accuracy of the 
diagnosis; and 7) Finances: the level of protection against hardship due to medical 
expenses and the extent of arrangements for getting help with financial problems. These 
seven categories have been defined as indicators of overall satisfaction by the Committee 
on Performance Measures and are used as a working definition for patient satisfaction. 


(NCQA 1997) 
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IV. METHODS 


A. 1996 DOD ANNUAL SURVEY FORM A 
The Department of Defense is required by Congress to conduct an annual survey 

of its active duty personnel, retirees, and eligible family members for the purpose of 
measuring accessibility, usage, and satisfaction in a range of health care services. The 
first of these surveys, referred to as the 1994-95 Health Care Survey of DoD 
Beneficiaries (HCSDB) was completed in the spring of 1995. This paper will analyze 
portions of Form A, the adult survey for 1996 HCSDB conducted by OASD(HA) Health 
Budgets and Programs. A point of contact may be reached at (202) 761-7895, Ext 246. 

1. Participants 

A sample of 156,838 beneficiaries was selected from a sampling frame of 6,455,915 
adult sponsors and family members 18 years or older from the Defense Enrollment 
Eligibility Reporting System (DEERS) of eligible beneficiaries based on current 
eligibility information as of October 28, 1995. The sample was stratified into four 
beneficiary subgroups: 1) active duty personnel, 2) family members of active duty 
personnel, 3) retirees, their family members, and survivors under age 65, and 4) retirees, 
their family members, and survivors, age 65 and over. (WESTAT 1996) 

2. Materials 

The survey is a nine-part multi-faceted questionnaire that covers the eight 
performance measurements outlined by the HEDIS requirements. The survey consists of 
84 questions covering the following sections: I) Your Health and Daily Activities; II) 


Preventive Health Care and Health Habits; II) Place of Medical Care and Health 


19 


Insurance Coverage; IV) Medical Care at Military Facilities; V) Medical Care at Civilian 
Facilities; V1) Dental Care; VII) TRICARE; VIII) Facts About You; and IX) Comments. 
This paper will primarily look at section IV, Medical Care at Military Facilities. Section 
IV consists of questions asking if the respondents had received any healthcare in the last 
12 months from a Military Treatment Facility (MTF). If the answer was “yes” then they 
were asked about the following issues: 1) length of stay in the hospital; 2) number of 
outpatient visits; 3) number of calls to obtain an appointment; 4) length of waiting time in 
the doctor’s office; 5) length of time to travel to the military or civilian facility; 6) 
waiting times between appointments and seeing the doctor based on acuity of care; 7) 
overall satisfaction; and 8) 32 aspects of satisfaction with the care the respondent 
received. (WESTAT 1996) 

3. Procedures 

The survey mailing involved four waves of mailings to beneficiaries between April 
15, 1996 and August 8, 1996. The multiple mailings were scheduled to maximize 
response rates for the data collection effort. First, an initial notification letter was mailed 
to the entire sample. Four weeks after the start date of the initial notification mailing, a 
first wave of surveys was mailed to the entire sample. Next, approximately two weeks 
after the first wave of survey mailings, a thank-you/reminder letter was mailed. Finally, 
four weeks after the mailing of the follow-up letters, a second wave mailing of surveys 
was directed to those who had failed to respond to previous mailings. The field was 
closed for targeted mailings on September 10, 1996. (WESTAT 1996) 

In order to avoid data analysis errors, several types of records were removed from the 


1996 HCSDB. These removals were performed when: 1) the same sample member 
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filled out more than one survey (if they received more than one mailing wave); 2) 
surveys were returned from sample members who were identified as deceased or 
incarcerated; 3) surveys were not filled out by the intended respondent; 4) surveys were 
blank and 5) there was a gender mismatch. If a sample member returned more than one 
survey, then the first wave survey was accepted and placed in the file. If the first wave 
survey was returned blank, then the second wave survey was selected. (WESTAT 1996) 

A final step was taken to ensure that the person completing and returning the survey 
was the same person the survey was mailed to. A survey record was judged to be in error 
if the gender of the sample member was different than the self-reported gender. In these 
cases, it was evident that someone other than the intended sample member completed the 
survey. Overall, 1,329 records indicated that someone other than the sample member 
completed the survey were removed. (WESTAT 1996) 

Of the 92,682 returned surveys, a total of 2,981 surveys were removed from the 1996 
HCSDB data file, yielding 89,701 surveys. The overall response rate was 58.18%. The 
response rates for the subgroups were: 45.0% for active duty ; 45.9% for family members 
of active duty; 68.5% for retirees under 65; 80.5% for retirees age 65 and older, 57.5% for 
family members of retirees under 65 years; and 72.8% for family members of retirees 65 
and older. (WESTAT 1996) 

B. ANALYSIS 

The 1996 Annual Survey is a complex survey that uses techniques such as probability 

sampling, stratification, clustering and weighting. The purpose in analyzing the surveys is 


to identify and understand the causal mechanisms and to investigate the links between 
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overall patient satisfaction and the seven factors defined by the HEDIS report card along 
with demographic variables such as sex, age, and beneficiary group. 

The first step in this analysis was to match the 34 questions from the patient satisfaction 
section of the Annual Survey to the NCQA questions found in the HEDIS Report card. 


Tables 1 shows the comparison of each of the DoD Annual survey questions with the report 


card questions. 


Table 1. Comparison of Questions 

HEDIS Report Card Questions 
OVERALL 
The overall health care provided through your health 
plan 
ACCESS: Arranging for and Getting Care 


Convenience of location of the doctor’s office 
Hours when the doctor’s office is open 

Access to specialty care if you need it 

Access to hospital care if you need it 

Access to medical care in an emergency 
Arrangements for making appointments for 
medical care by phone 

Length of time spent waiting at an office to see 
the doctor 

Length of time you wait between making an 
appointment for routine care and the day of your 
Visit 

Availability of medical information or advice by 
phone 

Access to medical care whenever you need it 
Services available for getting prescriptions filled 


TECHNICAL QUALITY 

a. Thoroughness of examinations and accuracy of 
diagnoses 

b. Skill, experience, and training of doctors 

c. Thoroughness of treatment 

QUALITY AND OUTCOMES 

a. The outcomes of your medical care, how much 
you are helped 

b. Overall quality of care and service 

COMMUNICATION 
Explanations of medical procedures and tests 


Attention given to what you have to say 
Advice you get about ways to avoid illness and 
stay healthy 


ID 


1996 DoD Annual Survey Questions 
OVERALL Q47a 
Q46a. I am satisfied with the health care that I 
received. 
ACCESS: Arranging for and Getting Care 
Q47 a-ff 
Convenience of location of treatment 
Convenience of hours 
Access to a specialist if you need it 
Access to hospital care if you need it 
Access to medical care in an emergency 
Ease of making appointments for health care by 
phone 
Length of time you wait at office to see the 
provider 
Length of time you wait between making an 
appointment for routine care and the day of your 
Visit 
Availability of health care information or advice 
by phone 
Access to health care whenever you need it 
Services available for getting prescriptions filled 


TECHNICAL QUALITY Q47]I-0 

l. Thoroughness of examination 

m. Ability to diagnose your health care problems 

n. Skill of health care providers 

o. Thoroughness of treatment 

QUALITY AND OUTCOMES Q47p-q 

p. The outcome of your health care (how much you 
are helped) 

q. Overall quality of health care 

COMMUNICATION Q47 r-u 
Providers explanation of health care procedures 
Provider’s explanation of medical tests 
Attention provider gives to what you have to say 
Advice provider gives you about ways to avoid 
illness and stay healthy 





Table 1. Cont'd 
HEDIS Report Card Questions 1996 DoD Annual Survey Questions 
INTERPERSONAL CARE INTERPERSONAL CARE Q47v-aa&dd 





























a. Friendliness and courtesy shown to you by w. Courtesy shown to you by health care providers 
doctors x. Provider’s concern for you as a person 

b. personal interest in you and your medical dd. Health care providers personal interest in the 
problems outcome of your problem 






Respect shown to you, attention to your privacy 
Reassurance and support offered to you by 
doctors and staff 

Friendliness and courtesy shown to you by staff 
Amount of time you have with doctors and staff 


during a visit 







y. Provider’s concern for your privacy 

z. Reassurance and support offered to you by health 
care providers 

v. Courtesy shown to you by administrative staff 
(e.g. receptionists) 

aa. Amount of time with health care providers during 

a visit 


as) 


rm 























CHOICE AND CONTINUITY 
a. Number of doctors you have to choose from 

b. Arrangements for choosing a personal doctor 
c. Ease of seeing the doctor of your choice 
FINANCES 


CHOICE AND CONTINUITY Q47bb-cc 
bb. Ability to choose health care providers 
cc. Ease of seemg the provider of your choice 








FINANCES Q47ee-ff 

a. Protection you have against hardship due to ee. Protection you have against financial hardship 
medical expenses due to medical expenses 

b. Arrangements for you to get the medical care you | ff. Help with arrangements to get the health care 

need without financial problems you need without financial problems 










1. Variable Selection 

The survey data consisted of 89,701 observations and 402 variables that included all of 
the survey questions responses and the constructed and scaled variables that were developed 
by the contractor, WESTAT, to support analysis of the data. Section IV of the questionnaire 
deals with beneficiaries’ satisfaction with health care they received at the military facilities. 
The satisfaction variables used in the analysis from the questionnaire are from questions 46a 
and 47a-ff pertaining to the military. The response for question 46a, concerning overall 
satisfaction, had a Likert scale with responses of: strongly agree, agree, neither agree nor 
disagree, disagree, and strongly disagree. These were assigned values in order from 5 to 1. 
The remaining questions (Q47a-ff) had a Likert scale with responses of: excellent, very 
good, good, fair, poor and not applicable. The individual responses for each of these 


questions were given a score from 5 down to 1 plus NA. Then the set of questions within 
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each of the seven HEDIS factors was averaged to obtain one value for each variable that 
described the essence of the outcome being measured. So, for example, the responses for 
the four questions concerning technical quality were converted to numbers on the 1-5 scale; 
then the average of those four numbers was used as the number for fechnical for that 
particular respondent. This type of manipulation of the data is acceptable as long as the 
original scales from excellent to poor are not referred to in the resulting conclusions 
(Agresti 1984). 

2. XPASTUSE 

From Section IV, a variable called XPASTUSE was constructed by the contractor, 
WESTAT, INC., to represent whether a beneficiary had used any military care (question 38) 
within the past 12 months. The XPASTUSE variable has 4 levels of use: 1=military only, 
2=civilian only, 3=both military and civilian and “.”=NA, no use of any medical facility 
within the past 12 months. All records with XPASTUSE=1 (military only) were extracted 
for use in the data analysis. 

3. Dependent And Independent Variable Selection 

The dependent variable overall was the response variable from question 46a, “I am 
satisfied with the health care that I received.” In order to use logistic regression and to 
obtain a clear distinction between dissatisfaction and satisfaction, overall was converted to a 
dichotomous variable with O=dissatisfied and 1=satisfied. Every value that was less than 3 
was set equal to 0 to represent dissatisfaction, and every value that was greater than or equal 
to 3 was set to 1 to represent satisfaction. For the trellis graphic display, the response 
variable used the original 5 factor levels of question 46a. The independent explanatory 


variables were the seven HEDIS averages access, technical, quality, communication, 
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interpersonal, choice, and finances and three demographic variables age, sex and 
beneficiary group. 
C. LOGISTIC REGRESSION 

The responses of survey questions are considered to be ordinal categorical data where 
there is a clear ordering of categories, but the absolute distances among them are unknown 
(Agresti 1984). However, ordinal responses are quantitative in the sense that each level on 
its scale can be compared in terms of whether it corresponds to a greater or smaller 
magnitude of each characteristic (Agresti 1984). Alan Agresti, in his book Analysis of 
Ordinal Categorical Data states that “The distinction regarding whether data are continuous 
or discrete is often less important than whether the data are qualitative (nominal) or 
quantitative (ordinal or interval)” (1984). Since this complex survey uses ordinal Likert 
scales for measurement of questions, logistic regression is an appropriate too! for data when 
a response variable exists as a binary categorical variable, and the explanatory variables are 
a mix of categorical and quantitative (Agresti 1984). 

1. Data Subsets for Analysis 

Three approaches were taken to using logistic regression in analyzing this complex 
survey. The first approach took a random sample from the large data set to look at a global 
view of patient satisfaction attitudes. One of the limitations of many statistical programs is 
their lack of ability to handle missing data. A large portion of respondents utilized the NA 
option on some of the questions 47 a-ff, and SPLUS 4.5, the data analysis software, omits in 
logistic regression all observations with any NAs. Next, a large random sample of 25,000 
observations was taken from the total of 89,701 observations without replacement to form 


the first tier of the data set. The data were then divided by the XPASTUSE variable into 


Z5 


three groups according to whether they had used medical treatment facilities in a military 
institution, or a civilian institution, or both. Logistic regression was used on the data set 
where beneficiaries had only used military healthcare. 

The second approach was to first sort the entire data sample by the XPASTUSE variable 
and extract all of the beneficiaries who had used military healthcare in the past year. The 
data were then sorted by region to look and see if there were any regional effects of patient 
satisfaction among beneficiary groups. Region 2, the Mid-Atlantic, was selected from the 
regions to be used as a representative in the analysis. Logistic regression was then used on 
the complete subset of beneficiaries who had used only military healthcare in Region 2. 

The third approach was to break the data sets down even furtner within Region 2, by 
beneficiary group, to determine if there were any differences among the variables for 
patients who have used only military healthcare. 

2. Procedures 

For each of the three approaches, a data frame was constructed with overall as the 
response variable, and as the explanatory variables, the seven HEDIS variables and the 
appropriate demographic variables. In addition to the dependent and the independent 
variables, a column of weights developed by the contractor, WESTAT, INC., was used in 
the analysis to account for the practical limitations of the survey such as imprecise survey 
frames and non-response by beneficiaries. In the initial models the response variable was 
modeled by the explanatory variables appearing only as the main effects. Then each of 
these was used in a stepwise selection process that permitted the addition of second order 


interactions. 
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D. TRELLIS GRAPHICS 

1. Explanation of Method 

Visualization of data sets is one of the most important techniques in statistical analysis. 
The Trellis Graphics Display framework, a recently-developed method for visualizing data, 
is based on a system of panels that look like a garden trellis (Becker 1998). While graphical 
visualization techniques have been available at least since the 1930s, the idea of Trellis arose 
from fees to study data structures with complex dependencies and interactions among 
the explanatory variables acting upon the response (Becker 1998). Trellising was first 
described in William S. Cleveland’s book Visualizing Data, written in 1993, which used 
multi-panel conditioning for ne to two conditioning variables (Becker 1998). Univariate, 
bivariate, and trivariate data can all be visualized in relation to a physical reality of space. 
But for any data set that has four or more dimensions, called a hypervariate data set, there is 
no way to visualize all of the variables together in the physical space (Cleveland 1993). 
Cleveland found that the solution to visualizing hypervariate data is to visualize it 
cognitively, not perceptually, through looking into the subspaces of the data (Cleveland 
1993). Since most common graphical approaches for visualizing data could only handle two 
explanatory variables, the need for visualization of three or more explanatory variables 
required the development of new graphical technology (Becker 1996). 

From 1993 to 1996 William Cleveland and Richard Becker developed and designed a 
visualization graphics method that has been implemented in the S-PLUS graphics to view 
many variables on a multi-panel display. The display panels are laid out into rows, columns 
and pages on which subsets of data are graphed and displayed using well-known graphical 


tools such as scatterplots, boxplots, normal quantile plots, and 3-D plots. Each panel shows 


2g, 





the relationship of a response variable to one or two independent variables, conditional on 
the values of some additional independent variables. (Becker 1996) 

The Trellis Display is a three-way rectangular array of panels with columns, rows, and 
pages that are read starting at the bottom of the page and going from left to right and then up 


the page, like a graph. Figure 1 below is a series of boxplots that has a response, 


explanatory and conditioning variables. 


Boxplots of Overall Satisfaction by Age, Regions and Beneficiary Groups 


20 40 60 80 100 20 40 60 80 100 


Satisfied Overall 





Age 


Figure 1. Overall Satisfaction Modeled by Age. 


The y variable is the response variable graphed on the y-axis of each panel, and the x 


variable is one of the explanatory variables, graphed on the x-axis. In this example, y is 
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overall satisfied and x is age. The conditioning variables, which are region and beneficiary 
group, are the other explanatory variables that are being held constant at a particular set of 
factor levels or for continuous-data, in a defined interval. The names of the conditioning 
variables are given in a strip label at the top of each panel. Here, the first conditioning 
variable is region and the second is beneficiary group. Each of these labels has a dark 
shaded bar in the same color as the strip label of the panel that indicates the value of the 
variable that is being held constant. The strip label and the bar show how the values of the 
conditioning explanatory variables are changing as they go throughout the trellis. The 
colors used in the assembly of the panels are analogous colors that aid in the decoding of the 
picture. Insight and understanding of the data structure is found by examining the 
relationships between the response and the conditioning variables. (Becker 1996) 

There are four key elements which make the Trellis Graphics Display different from 
other graphics methods. The first is the “Main-Effects Ordering.” This display method 
allows the ordering of the response variable by median, mean, or any special ordering that 
may be naturally occurring in the data. At the same time, the conditioning variables may 
also be ordered to obtain a natural flow of data that can easily be visualized. Figure 2 shows 
a graph of Region 2 with the HEDIS variables and beneficiary groups in alphabetical order. 
(Becker 1996) 

It is difficult to read this graph quickly and to recognize the variables which are rated 


highest and lowest and the variables for which the sexes differ. 
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Means of HEDIS Measurements for Region 2 


Figure 2 Alphabetized Factors. 


Figure 3 shows how the main effects ordering property makes the visualization of the 
data much easier to read and to quickly see the differences between the variables. In this 
figure, the HEDIS variables and the beneficiary groups are reordered from lowest median to 
highest median. It is now much clearer that active duty beneficiaries have a lower median 
overall than the other groups and the HEDIS variable choice has been rated distinctly lower 


for all beneficiary groups and overall has been rated distinctly higher by each beneficiary 


group. 
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on more than two variables. 


Region 2 Satisfaction Scores 





Means of HEDIS Measurements for Region 2 


Figure 3 Main-effects Ordering by Median. 
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o 


Male 
Female 


The second element that distinguishes Trellis from other graphics methods 1s the control 
of the “aspect ratio.” The aspect ratio the height/width ratio of the data rectangle, that is the 
panel rectangle that encloses all of the data. Graphical theory has shown that the banking of 
45° gives the best perception of relative slopes and curvatures of a line (Becker 1996). 
Because this is an important part of graphing, the Trellising system gives the users direct 


access to define the aspect ratio. Figure 3 is the same picture as Figure 2, but banked by 


The third element that makes this method extremely valuable is the ability to condition 
The ability to arrange the variables allows control of the 


explanatory variables while viewing multiple comparisons of many combinations of 





variables. Figure 4 shows the histograms of “age” while the regions move from left to 
right and the beneficiary groups move from bottom to top. This ability to condition on 
more than two variables allows further investigations of many different demographic 
subsets. 


Distribution of Age by Region and Beneficiary Group 


20 40 60 80 100 20 40 60 80 100 


Percent of Total 





Age 


Figure 4. Conditional Variables. 


Lastly, the Trellis software allows for conditioning variables to be continuous. It has a 
function called shingle() that allows the user to divide a variable into overlapping intervals, 
each with the same number of points, to use continuous variables as conditional variables 
(Becker 1996). This is extremely important for uncovering interactions between 


explanatory variables since it allows the conditioning variable to be held nearly constant. 
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2. Data Subset 

Regions 1, 2, 9, and 11 were selected from the fourteen regions based on their location. 
These included the three Naval teaching hospitals, The National Naval Medical Center in 
Bethesda MD, Naval Hospital Portsmouth in VA, Naval Hospital San Diego in CA, and a 
Fleet Hospital, Naval Hospital Bremerton in WA. These regions were selected to use in 
studying the interactions among variables and for showing the HEDIS variables that may 
help military medical facilities provide better care for their patients. 

3. Procedures 

In order to visualize the relationships among variables in the four regions these steps 
were taken. The first step was to see an overall view of what the data looked like. To do 
this, the histogram was first used to display the densities of the different ages among 
beneficiary groups and sex. This is important to see because age was not used as a 
conditioning variable since beneficiary groups provide decision makers with a better 
demographic breakdown based on the levels of access outlined in Title 10, U.S.C. Chapter 
55, Sections 1071-1106 which describes the entitlement programs. Next, the satisfaction 
scores for the four regions were shown to determine if there are any areas among the HEDIS 
factors that the primary military facilities in each region can use to provide better care within 
each subset of beneficiaries. This was done by calculating the means of each HEDIS factor 
or each conditioning variable and plotting them together to show comparisons and to see an 
overview of the data for each of the four regions. 

The last approach investigated the complex interactions and dependencies among the 
HEDIS variables and the demographic variables. Trellis was used to model four variables 


at a time and then used to investigate even more complex interactions with five variables. 


This step used boxplots to summarize the conditional distributions. In a boxplot, the dot 
represents the center of the data or the median of the distribution. The upper and lower ends 
of the box represent the 25" and 75™ percentiles and therefore the distance between the ends 
of the box is indicates how tightly packed the values are around the median (Cleveland 


1993). 
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M. RESULTS 
A. LOGISTIC REGRESSION 
The first approach outlining the methods was to take a random sample of 25,000 from the 
data set of 89,701 responses. After subsetting the data for military use only and accounting 
for missing, data, the model used a sample size of N=4,361 with 27 explanatory variables. 
The initial main-effects generalized linear model that included regions showed all of the 
variables, except the dummy variables for regions 6 and 12 to be significant. Table 2 shows 


the results of this approach. 


Table 2 Logistic Regression of Main-effects Model with Regions. 


Coefficient Value Sta. Error t value 
(Intercept) -4.088 0.034 -117.206 
access 0.628 G-002 64.550 
technical 0.196 Ox.0 aun 8.477 
quality 0.800 OFUrT (e953 
communication -0.145 CG. 010 -13.879 
interpersonal 0.340 0.010 32.394 
choice Gipies aa O2007 45.566 
finances -0.147 02005 -28.116 
Ben .Group2 Cras Chee Oe Zico 
Ben .Group3 0.434 OP (ans, 22.165 
Ben .Group4 Rien Oe, Ob (GSS, T82526 
Age 0.009 Cz000 14.650 
REGION2 -0.619 0.019 =31%:759 
REGION3 0.264 O2021 ee 
REGION4 0.210 0.026 7.883 
REGIONS 0.2326 0.024 13.042 
REGION6 -0.014 OP O)aes, -0.732 
REGION7 0.068 0.026 2.543 
REGIONS O..3:.10 GA0Z22 13.754 
REGIONS OS 757 Om023 S32 3562 
REGION10 Uses CZUS6 10.999 
REGION1I1 0.407 O03 3 3S 0 
REGIONi2 -0.043 O3035 =e 22 3 
REGION13 0.186 0.048 3.830 
REGION14 0.136 0.019 6.927 
sex -0.234 O2012 -18.079 


One of the troubling results found in the main-effects model and found in the 2" order 


interaction model was a negative sign for the coefficients of variables such as 
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communication, and finances. The negative correlation implies that the patients who rated 
overall satisfied high, were less satisfied than those who rated overall satisfied low for those 
variables. Similar counter-intuitive results were also found in the 2™ order interaction 
models. 

The above models were very complex and too large to explain with clarity. The data was 
broken down further into regions because of the scores of interactions. Region 2 with a 
sample size of 1351 was selected for this analysis. The results for the main-effects model 
and the 2" order interaction model were similar, showing negative values for the 
coefficients of the main variables such as finances, communication and interpersonal. 

As a final step, the data mas broken down by beneficiary group to see if there were any 
differences among the types of beneficiaries that would lead to interactions and negative 
coefficients. Region 2 was divided into beneficiary groups with the following N: active 
duty N=493, active duty family members N=268, retirees and their family members <65 
years old, N=390, and retirees and their family members >= 65 years old, N=200. The 
results, using stepwise regression, from the main-effects model by beneficiary group showed 
variations between each group. In Table 3, the active duty group added ‘technical as a 
negative coefficient but still had all of the variables as significant. 


Table 3 Active Duty main-effects model. 


Coefficient Value Staeeanror t value 
(Intercept) -8.023 0.067 -119.455 
technical -0.651 C020 -31.487 
access 0.687 O2016 Ae 227 
finances 0.419 0.009 -42.137 
choice 0.654 OO 48.826 
interpersonal 0.407 0.018 2h e4 
communication -0.312 0.019 -15.868 
quality 2.768 0.023 115.819 

sex 0.364 0.027 13.288 

age O.202.b O.2008: 26.751 
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In Table 4, the active duty family member group found fimances and access to be 


insignificant. 
Table 4 Active Duty Family Members main-effects model 
Coefficient Value Std. Enrer t value 
(Intercept } ~4,974 0.090 -54.264 
technical 0.439 0.036 12.063 
access -0.009 ONUZ9 -0.316 
finances U200s 0.014 0.270 
choice 0.637 0.020 31.438 
interpersonal 150.36 O.203 5 29.014 
communication -0.689 02027 -24.811 
quality I eehow 0.035 Be 1 ol 
sex -0.723 0.046 -15.682 
age = Od 0.001 ~7.774 


In Table 5, the model of the retirees and their family members who are younger than age 
65 showed all of the variables to have significance except interpersonal and did not have 


any negative correlation for the variables that were significant. 


Table 5 Retirees and Family Members <65 main-effects model 


Coefficient Value Std? Error t value 
(Intercept) -8.001 0.154 -51.927 
technical 0.989 Oe uSZ Seal Oi, 
access 0.756 0.043 7eS27 
finances OeizZy 0.024 5. 300 
choice 0.475 OF ()8) 14.283 
interpersonal -0.019 0.046 ~0.421 
communication OmolT 0.048 8.431 
quality 0.564 0.046 12.260 
Sex 0.450 0.045 wap’ Sy 
age 0.016 0.001 8.847 





And lastly, Table 6 gives the main-effects model for the retirees and their family 
members who are age 65 years and older. It showed technical and access as non-significant 
and had negative correlation for finances, choice, and communication with the response 


variable overall. 


oy 


Table 6 Retirees and Family Members >= 65 main-effects model 


Coefficient Value Sea. Erron t value 
(Intercept) -15.982 e627 ~9.820 
technical Ome 0.214 Ore Oa 
access O27 1503 OFa alls O2563 
finances 07275 OF 15 -2.382 
choice -0.496 0.268 =2 2356 
interpersonal 3.439 Om337 Hoe 8 
communication -0.918 0.285 -3.218 
quality 2.245 02375 Tella 

sex 5.242 0.411 12.749 

age 0.048 O02 1 2.52. 


The 2” order models by beneficiary groups were similar to the first two approaches with 
many combinations of significant interactions between explanatory variables. 

The three approaches above used logistic regression to predict overall patient satisfaction 
by the seven HEDIS variables and the demographic variables. Each primarily showed that 
almost all of the variables are important with some variables such as finance and 
communication had some negative correlation with the response variable overall. In 
addition, there were complicated interactions in each the data sets among the demographic 
variables and the HEDIS measurements. One of the Trellis Graphics display method’s 
strength is its ability to uncover interactions through visualization and conditioning of 
variables. 

B. TRELLIS GRAPHICS 

1. Age Distributions of Regions 1, 2, 9, and 11 

One of the beneficial features of Trellis is the ability to view distributions of 
subgroups at a glance. The analysis of the patient satisfaction data set used the subgroups 
region, beneficiary group and sex. Although age was one of the initial demographic 
variables, by definition, the beneficiary group delineation provided an idea of the range 


of ages as shown in Figure 5 for female respondents and Figure 6 for male respondents. 
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Distribution of Age by Region, Beneficiary Group, and Sex 


Percent of Total 





Age 


Figure 5. Histogram of Female ages by Beneficiary Group. 


In row 1 of Figure 5 there is only a small number of female active duty respondents in 
each region. Figure 6 shows a comparison of the distribution of ages of the male 
respondents for the given regions. Active duty males had the largest population of 
respondents in each of the regions; in each case the smallest group was male family 
members of active duty personnel. The two retiree groups had approximately the same 
distribution for both males and females with the ages of the younger population mostly 
falling within the 40-60 year old range and the age of the older group falling within the 


60-80 year old range. 





The number of returned surveys within the four regions are different with Region 1, 
the Northeast, which includes the National Capital area, being the largest. Region 11, the 
Northwest, has the smallest response rate and Region 2, the Mid-Atlantic, or Portsmouth, 
VA area and Region 9, Southern California, or San Diego area having about the same 


distribution of sizes. 


Distribution of Age by Region, Beneficiary Group, and Sex 
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Figure 6 Histogram of Male Ages by Beneficiary Group. 


Zz Dotplots of HEDIS Measurements 
In order to help military medical facilities locate key issues to improve healthcare for 
their patients, the different scores for the HEDIS measurements should be analyzed for 


each region. The dotplot of the mean scores is one of the best techniques for providing a 
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highly accurate visual decoding of data (Becker 1996). First, the mean for each of the 
HEDIS variables was calculated by region, beneficiary group, and sex. Then Trellis 
graphics was used to plot the means of each measurement by HEDIS variable, region, 
and sex in an easy-to-read picture. Figure 7 below shows all four regions ordered by their 


overall medians. 


Regions 1,2,9 and 11 Satisfaction Scores 
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Means of HEDIS Measurements 


Figure 7 Regions 1, 2, 9, and 11. 


The choice variable has the lowest median overall, and the variable overall has the 
highest median. Panel (1,1) in the lower left corner shows that Region 2 has the lowest 
median and in panel (4,4), retirees and their family members over 65 years old has the 


highest median. 
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The variable choice was rated the lowest by each beneficiary group in each of the four 
regions with the exception of male active duty family members rating finance lower in 
Region 11. It is also interesting to note that the women in Regions 11 and 1 are very 
different from the men. There seems to be an upward trend from active duty 
beneficiaries to retirees, indicating that the older population tends to be happier than the 
younger groups. These relationships will be examined in greater detail as each of the 
four regions is discussed individually. 

Grouping the regions together on one graph provides a comparison between 
regions. However, it is best to graph them separately; that way the greater spread of the 
values can better show distinguishing features of the satisfaction scores. Region 2’s 
satisfaction scores are shown in Figure 8. 

The variable choice here is shown on a finer scale than in Figure 7 and demonstrates the 
difference of ratings by the different beneficiary groups. For active duty members (females: 
N=72, males: N=421) and their families (females: N=246, males: N=22), women gave 
choice a lower score than the men. In addition, for all other factors except finances for the 
active duty members and technical for the family members, the women gave lower ratings 
than the men. For retirees and their family members under 65 (females: N=199, males: 
N=191), there was no distinct separation between the sexes. Finances have the second- 
lowest ratings for the retiree groups. Overall, the older beneficiaries seemed to be happier 


with military facilities than active duty members and their families. 
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Region 2 Satisfaction Scores 
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Figure 8 Region 2. 


Region 11, shown in Figure 9, had the second lowest overall median score. Region 1 1 
was very different from the other regions primarily in that there is a clear distinction 
between men and women for their satisfaction scores. There was almost a full point 
difference between males and females in almost every category for each beneficiary group. 
The most striking difference was for active duty family members who rated the factor 
finances. However, on closer examination, the small number of respondents in the male 


family member group (N=8) may somewhat overstate the difference compared to the larger 
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group of females (N=148). 


Region 11 Satisfaction Scores 
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Means of HEDIS Measurements for Region 11 


Figure 9 Region 11. 


The overall satisfaction score for the retirees and their family members age 65 years and 
older was a deviation for men (N=55) from the dominant trend. In this group, overall is 
rated lower than the next four highest factors, interpersonal, technical, quality, 
communication, and access. If overall is thought to be approximately an average of the 
other variables, then this was the only group that indicated that result. With the exception of 
this one value, the spread from the beneficiaries’ ratings from all other variables to overall 
was very large which is somewhat an anomaly. The last noteworthy finding is that given the 


low number of active duty family members who are male, females across all of the 
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beneficiary groups in Region 11 were almost always less happy than males. This was 
shown for every HEDIS measurement except finance for the active duty groups and for 
overall satisfied for retirees and family members younger than age 65. 

Region 9 had the second highest median score of the four regions as shown in Figure 10. 
There are two strikingly different findings for Region 9 that were distinct from those in the 
other regions. First, as shown by the ordering of the colors in the key, males had the lower 
median response rate. In all of the other regions, females had the lower median. 

The second finding is that Region 9 was the only region where active duty males 
(N=463) rated choice lower than females (N=67); here the difference in the spread of ratings 
is approximately a full point. This region was also the only one in which the active duty 
men were predominantly less happy than the women. This is also seen for the active duty 
family members who are male: though there are only 14 men in this group; when put with 
the 463 active duty males who felt similarly, this is a substantial finding. 

Additionally, the overall marks were also very high in this region which was not what 
would be expected with such a score. It is also interesting to note that even though the 
males in the two active duty groups were less happy in almost all of the HEDIS 


measurements, they still rated overall (satisfied) higher than the females in Region 9. 
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Region 9 Satisfaction Scores 
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Means of HEDIS Measurements for Region 9 


Figure 10 Region 9. 


The National Capitol Region had the highest median among the four regions. Figure 11 
shows the satisfaction scores for Region 1. Within this region the variables choice and 
overall stand out and follow a trend similar to that in the other regions. However, the spread 
from active duty members (female: N=128, male:N=743) to retirees and their family 
members age 65 and older (females: N=204, males: N=210) for choice is very wide. For 
retirees and their family members younger than age 65, the males (N=296) were more 
unhappy than the females (N=272) for the measurements finance and choice. Also the 


spread between the sexes in all of the beneficiary groups except active duty members is very 
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noticeable indicating different attitudes and feelings among men and women respondents in 


Region 1. 


Region 1 Satisfaction Scores 
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Means of HEDIS Measurements for Region 1 


Figure 11 Region 1. 


3. Trellis Interactions 

Interactions among variables of a multivariate data set occur when there is 
interdependence among variables at different levels or when the effect of a variable depends 
on the values of the other explanatory variables. The logistic regression models all showed 
complex interactions occurring among the HEDIS variables, between the HEDIS variables 


and the demographic variables, and between the demographic variables region, sex and 
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beneficiary groups. Interactions can cause difficulties in using logistic regression to predict 
a response variable when they are too numerous to explain causality clearly and concisely. 
If a data set has explanatory variables that have strong dependencies among themselves, like 
the HEDIS measurements, which are attitudes of respondents, then interactions among 
variables are likely and can be difficult to identify. The Trellis Graphics framework is 
extremely helpful in seeing interactions and viewing complex data sets. 

Figure 12 is a graph with boxplots of overall satisfaction modeled by access while 


holding each region and beneficiary group as conditional constants. 


Boxplots of Overall Satisfaction and Access by Region and Beneficary Group 
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Figure 12 Relationships of Variables with Few Interactions. 
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This figure demonstrates a set of variables that have few or no interactions among 
variables. Interactions are seen as different slopes of panels as the eye moves through the 
conditional variables. In row 1 of Figure 12, as the eye moves from left to mght, there seems 
to be very little interaction occurring between regions. Each region has approximately the 
same shape and slope of the medians (dots within the boxplots) and the sizes of the boxes 
are about the same, indicating similar distributions of the interquartile ranges. When 
looking at a specific region such as Region 1, panel (1,1) and panel (1,2) appear to be 
approximately the same, indicating little interaction or interdependency within Region 1 
among beneficiaries between beneficiary groups in the first two groups and the relationship 
between access and overall satisfaction. \dentifying groups of variables that do not have 
interactions may help the decision-maker understand relationships which can be explained 
relatively easily. For example, as access increases, the rating of overall satisfaction by 
respondents in Region 1 who are active duty or family members of active duty will increase 
by approximately the same amount. In other words, if a military facility would like to 
increase overall satisfaction, then there is consistent relationship between that variable and 
ACCESS. 

In the 1994-95 survey, access was rated as the lowest measurement indicating it was 
one of the most dissatisfying aspects of care for beneficiaries. For the 1996 survey, choice 
was rated the lowest and access moved to the third-lowest measurement, pointing to some 
improvement. The finance measurement was rated as the second lowest measurement on 
the 1996 survey. The measurement consists of two questions: 1) “Protection you have 


against financial hardship due to medical expenses” and 2) “Help with arrangements to get 
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the health care you need without financial problems.” Figure 13 is the graph of satisfied 


overall modeled by finance conditioning on regions and beneficiary groups. 


Boxplots of Overall Satisfaction and Finances by Region and Beneficiary Group 
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Figure 13 Relationships of Variables with Interactions. 


This picture is very unusual because in almost every panel by region and beneficiary 
groups, the finance measurement has a median of 3. The response by active duty members, 
who should feel that the military will cover their own medical expenses was expected to be 
higher than 3. Instead, this group has several of the ratings across regions that were lower 
than the median value of 3. This could indicate that when thinking of financial security, 


these active duty members might be describing their security as it relates to their families, 
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not as individuals. In column 3, which is Region 2, there are some interactions occurring 
between the beneficiary groups, while there appear to be only minimal interactions 
occurring in Region 11. 

Figures 14 and 15 are graphs of overall satisfaction modeled by the choice measurement 


by region, beneficiary group and by sex (Figure 14: female, Figure 15: male). 


Boxplots of Overall Satisfaction and Choice by Region, Beneficary Group and Sex 
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Choice Measurement 


Figure 14 Interactions among Choice Variable for Females. 
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Boxplots of Overall Satisfaction and Choice by Region, Beneficary Group and Sex 
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Choice Measurement 


Figure 15 Interactions among Choice Variable for Men. 


The choice variable had the lowest mean satisfaction score of all of the HEDIS 
measurements. The variable is defined by the questions: 1) “Ability to choose health care 
providers” and 2) “Ease of seeing the provider of your choice.” This variable relates to 
what are really two separate issues, choice of provider and continuity of care. There are 
many interactions going on between beneficiary groups as seen, for example, by starting at 
the bottom of each column and going up, holding the region constant. This is also true 
among the different beneficiary groups across regions. In Figure 14, there are very few 
females who rate choice above the value 3. The active duty females seem to be very 


unhappy in Regions 2 and 11 and only slightly happier in Regions | and 9. 
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4. Complex 5-Way Interactions 

Figures 16 through 19 are a set of 4 pages for a 5-way interaction with overall satisfied 
as the response variable. Choice is the continuous explanatory variable and the conditional 
variables are communication, region and beneficiary group. Each of the four figures 
represents a beneficiary group beginning with active duty members. . Below is Figure 16, 


which shows the active duty members for each region. 


Overall Satisfaction and Choice by Communication, Region, and Beneficiary Group 
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Choice Measurement 


Figure 16 5-Way Interactions for Active Duty Members. 


Each row on each page shows how choice is modeled by overall satisfaction holding 


communication constant for one-point intervals. Each column on each page is a comparison 
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of regions for each level of communication while choice and overall satisfaction are the 
response and explanatory variables. The level of interaction on these four figures are very 
complicated and difficult to interpret; however, Trellis provides the ability to see 
movements within the conditional variables which may lead to insights of multi-level 
interactions among variables. 

In Figure 16, columns 1 and 2 depict the first two levels of communication as is shown 
by the shaded portion of the strip label. As each level of communication progresses up one 
interval, interactions are occurring within each region. Figure 17 shows the interactions for 
active duty family members. There appear to be interactions for all levels of communication 
except the 4 to 5 intervals for communication in column 4. The slopes seem to be 
approximately the same although there are overall satisfaction levels that do not have any 
observations such as in Region 11 and Region 9. Active duty family members’ responses 
also appear to have rated choice lower than the corresponding levels of communication 
which is shown by the ratings in almost all of the panels being to the left of the shaded bar 
that indicates what level of communication is being measured. This shows the same results 


as the dotplots which is to show that choice is rated lower than communication. 





Overall Satisfaction and Choice by Communication, Region, and Beneficiary Group 
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Choice Measurement 


Figure 17 5-Way Interactions for Active Duty Family Members. 


Next, Figure 18 of retirees and their family members less than 65 year old, shows 


interactions throughout the page, with one possible exception of the slopes of the 


commumication interval from 2 to 3, which look similar in column 3. 
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Overall Satisfaction and Choice by Communication, Region, and Beneficiary Group 
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Choice Measurement 


Figure 18 5-Way Interactions for Retirees and Family Members < 65. 


The final beneficiary group in this example showing multi-panel interactions is the 
retirees and their family members who are age 65 years and older. This group had the 
highest median of overall satisfaction, reflected by the small number of observations in 
column 1 of Figure 19. But even in this group there are still the ratings of overall 
satisfaction given choice that are lower than the defined interval for communication. 

These four graphs of 5-way interactions are difficult to interpret unless there is some 
expert who is knowledgeable about the specific beneficiary groups within each region. In 
this type of situation, the complex interactions could provide more insight into the attitudes 


that are expressed through the survey. 


56 





Overall Satisfaction and Choice by Communication, Region, and Beneficiary Group 
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Figure 19 5-Way Interactions for Retirees and Family Members >= 65. 


5. Overview of all 14 Regions 

Figures 20 and 21 provide a comparison of the satisfaction scores for all 14 regions. 
In Figure 20, the two groups that have the lowest median are active duty members and 
active duty family members. The region having the lowest median is Region 3 and the 
region that has the highest median is Region 5. The general trend of women being less 
satisfied with healthcare is carried through most of the regions. However, there is one 
exception to this for one of the beneficiary groups in Region 5. Active duty males 


appear to be noticeably less happy than the females as shown in panel (7,2). Choice 
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remains as the lowest measurement across all regions and active duty females in Region 


12 are shown in panel (5,1) as being the most dissatisfied. 


Regions 1:14 Satisfaction Scores 
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Figure 20 Regions 1:14 Active Duty and Their Family Members Groups. 


Figure 21 shows that the beneficiary groups that have the highest median for all of the 
regions are the two retiree groups. Here the same general trends are seen as before with 
the measurements choice and overall satisfied. Retirees across all regions are much more 
satisfied with healthcare than active duty members and their families. 

The explanation for this occurrence may be due to the self-selection of retirees and 
their family members in wanting to have access to the Military Treatment Facilities. 


These two groups of beneficiaries have the least availability and access to care; when 
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they do gain access, they are likely to be extremely satisfied with that episode of care. In 
contrast, active duty members are directed to go primarily to the Military Treatment 
Facilities and do not generally have a choice in the location of where they may receive 
medical care. Active duty family members do have a choice in selecting the location of 
care, but may be constrained by financial concerns in those regions that did not have 
TRICARE, or may be unhappy with their situation in those regions that did have 


TRICARE. 
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Figure 21 Regions 1:14 for Retirees and their Family Members 
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VI. CONCLUSIONS 
A. DISCUSSION 

The primary objective of this paper was to determine if any of the HEDIS 
measurements would indicate areas among patient satisfaction scores that could be used 
by decision-makers to improve patient satisfaction. In addition to understanding these 
seven areas, another important question was whether there are differences in satisfaction 
between men and women or between the four beneficiary groups. 

First logistic regression was used to model overall satisfaction by the seven HEDIS 
variables and by the demographic variables used to describe the respondent population. 
Although this technique provided insight into the complexities of the data, it raised more 
questions than answers. One could just conclude that all of the variables and interactions 
among variables are significant. The problem with this approach is that just determining 
that all of the variables are important, indicated by the largest coefficients, does not 
describe the data properly or explain the negative coefficients for the main-effects 
models. The relationships among the variables were strongly interdependent and to 
understand them and how to improve patient satisfaction, that assumption is not enough, 
so an additional analytical approach was needed. 

The Trellis Graphic framework provided an excellent tool to visualize and understand 
the complex relationships between variables and the interactions of variables. One of the 
most important findings is that overall satisfaction is not an accumulation of the seven 
HEDIS factors because the overall satisfaction measurement was rated consistently 
higher than any other variable. It was expected that overall satisfaction would be 
approximately an average of the variables that define it. A possible explanation could be 


related to the positioning of the question. The question concerning overall satisfaction 
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with the health care appeared in the survey before the specific questions concerning 
access, technical, quality, choice and continuity, communication, finances, and 
interpersonal skills. There seems to be a design effect created by the order of the 
questions on the survey that leads respondents to answer to this early question more 
positively than to the other questions that make up the HEDIS variables. If a decision- 
maker were to use only this score as an indicator of patient satisfaction, his or her 
audience could potentially be mislead. 

The results of the Trellis graphics for the four defined regions suggests that across 
beneficiary groups the measurement that can have the greatest impact on improving 
patient satisfaction is choice, hich includes continuity of care. In a study conducted for 
Kaiser Permanente of Northern California, it was found that choice in choosing one’s 
Own primary care manager was the single most important factor in overall satisfaction 
(Greene 1998). This analysis would indicate military beneficiaries are not much different 
from their counterparts in the civilian health plans in regard to wanting to choose their 
own doctor and have continuity of care. Active duty members rated choice in the lowest 
ranges of the scale, indicating their extreme unhappiness with their ability to choose their 
provider. In the past, there has been anecdotal evidence to suggest that there is a feeling 
among, active duty members that they have less choice than dependents and retirees do. 
This analysis clearly indicates this perception by the active duty population through their 
low ratings of choice in every region. 

One answer to this problem might be the TRICARE program. TRICARE was 
developed as a combination of the strong points of the civilian sectors’ managed care 


plans to provide patients with better access, choice and quality of care. Under 
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TRICARE, active duty members are automatically enrolled in TRICARE Pnme, which 
means they are mostly enrolled at the military treatment facilities. Regions 9 and 11 had 
substantial experience with TRICARE at the time this survey was conducted but Regions 
1 and 2 had not begun the TRICARE program. There did not appear to be a consistent 
difference in ratings between regions that had TRICARE and those that did not, possibly 
indicating that TRICARE does not have an effect on how patients are managed within the 
military treatment facilities. However, further studies would be needed to examine the 
degree of interaction between these variables. 

In response to the question of whether there are differences in satisfaction between 
men and women, the answer is yes. There are some exceptions such as in Region 9, 
where active duty males were less satisfied in almost every measurement than females, 
but the primary trend was that women are less happy than men across beneficiary groups 
and regions. This was particularly true for Regions 1 and 11. Research done by 
Gregory Pascoe (1983) indicated that in general females tend to be more happy than men, 
which is opposite of the findings for these four regions for military treatment facilities. 

Studies have shown that older patients tend to be happier than younger patients, and 
this analysis agrees with those findings. The oldest beneficiary group, retirees age 65 and 
older was the most satisfied group across all regions. This was not what was initially 
expected since this group of beneficiaries fall under Medicare and have the lowest level 
of accessibility to military treatment facilities. However, since they have been able to 
gain access, they seem to be extremely happy with the care they have received, possibly 
because the healthcare is perceived as high quality while requiring little or no out-of- 


pocket expenses. 
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On a final note, this analysis and the resulting conclusions would not have been 
possible without the use of the Trellis Graphics framework. This graphing technique 
provided a new way to look into hypervariate data and break the barrier of graphing for 
up to three dimensions. This would have been impossible only a few years ago. This 
technique will revolutionize the way multivariate data sets can be used when other 
statistical techniques are limited by interdependencies among variables. Trellis Graphics 
can be especially useful in understanding complex behavioral surveys, like the DOD 
Annual Survey, that typically have discrete ordinal scales. Trellis’s strength lies in its 
seemingly simple, easy to read graphs that make the analysis of complex data 
understandable. This method was extremely useful in comparing multiple demographics 
of subgroups of beneficiaries that enabled additional insight into trends among HEDIS 
measurements for age and sex. Real-life data measuring attitudes and feelings often have 
complexities and interdependence that restrict the use of statistical tools such as logistic 
regression. The Trellis graphics allowed the visualization of the interactions that the 
logistic regression models were capturing but not explaining. 

B. RECOMMENDATIONS 

The military treatment facilities are the primary location of care for active duty 
members, yet this beneficiary group was the least satisfied for almost every HEDIS 
measurement. This indicates that there is a perception among active duty members, not 
held by the retiree groups, that health care in the military treatment facilities is 
inadequate. Since the active duty group is the primary customer of military treatment 
facilities, it is recommended that further studies be conducted to pinpoint the reasons for 


these perceptions. 
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The second recommendation is to include a question on overall satisfaction at the end 
of the 32 HEDIS measurement questions identical to the one at the beginning. This may 
allow for a different measurement of overall satisfaction and would also measure the 
effect of asking the question twice to see if there is a large variation in responses caused 
by the intervening HEDIS measurement questions. 

The third recommendation is the delete the NA options on the 32 questions on patient 
satisfaction. By providing that option, analysis of the questions 1s affected because many 
Statistical tools handle them by deleting each response with any NAs. This has the effect 
of decreasing the number of responses by up to approximately 50%. 

The final recommendation is to explore the use of Trellis Graphics for individual 
catchment areas. Valuable information can be obtained from seeing and understanding 
the responses of patients who have used a specific hospital; this would provide 


information and understanding to decision-makers at the lower levels. 
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